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SUMMARY 


THIS  s»vt».v  was  conducted  to  deteralne  If  aora  accurstt  trsnsforaatlons  of  digital  ttrraln 
data  Mould  significantly  Incrasst  tha  parcalvad  laaga  quality  or  training  effectiveness  of 
slaulatad  radar  laagary.  Mhlla  an  Incraasa  In  accuracy  produces  a  aora  detailed  laaga.  It  also 
requires  aora  coaputar  tlaa  to  generate  and  therefore  Increases'  the  cost  of  database  developaent. 

Seven  KC>136  navigators  evaluated  slaulatad  radar  laages.  generated  at  six  different  levels 
of  transforaatlon  accuracy,  for  usefulness  In  navigation  and  for  training  navigators.  Analyses 
sHon  that  trsnsforaatlons  which  are  aora  accurate  than  current  standards  do  not  produce 
perceptible  Increases  In  laaga  quality  or  In  rated  training  effectiveness.  Before  final 
conclusions  are  drawn,  however,  this  study  should  be  replicated  using  a  larger  saaple  of 
navigators  with  low  altitude  requlreaents  and  slaulatad  laages  froa  lower  altitudes. 
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PREFACE 


This  project  mi  conducted  In  support  of  the  Air  Force  Human  Resources  laboratory's 
Technical  Planning  Objective  3,  Aircrew  Training.  The  goal  of  this  effort  Is  to  develop 
cost-effective  strategies  and  equipment  for  aircrew  training.  This  experiment  was 
conoucted  under  Nork  Unit  6114-33-01,  Analysis  of  Imegery  for  Evaluation. 

The  goal  of  this  experiment  was  to  determine  If  Increasing  the  digital  terrain 
transformation  accuracy  would  Increase  the  apparent  Image  duality  and  perceived  training 
effectiveness  of  simulated  radar  Images.  Results  show  that  Increases  In  transformation 
accuracy  over  current  standards  did  not  produce  noticeable  changes  In  either  the  quality 
or  the  perceived  training  value  of  almuleted  ground  mapping  radar  Images.  Simulated 
Images  produced  with  less  accuracy,  however,  ware  judged  to  be  of  poor  quality  and  not 
usaful  for  training  navigators.  Therefore,  the  current  standard  for  vertical 
transformation  accuracy  represents  an  appropriate  compromise  between  cost  end  training 
effectiveness.  However,  operational  navigators  who  served  as  subjects  recommend  that 
this  study  be  replicated  with  lower  altitude  Imagery  before  drawing  final  conclusions. 

The  authors  wish  to  thank  John  Stengel  of  the  Aeronautical  Systems 
Dlvltlon/Englneerlng  for  providing  the  stimuli  and  the  navigators  of  the  161st  Air 
Refueling  Group,  Arizona  Air  National  Guard,  who  participated  as  subjects. 
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FLIGHT  TRAINING  SIMULATORS:  EFFECTS  OF 
TERRAIN  ACCUP.ACi  ON  SIMULATED  RADAR  IMAGE  QUALITY 


I.  INTRODUCTION 

This  preliminary  experiment  was  conducted  to  deterelne  If  a  more  detailed  transformation  of 
digital  tarraln  data  would  significantly  Incraata  tha  training  erfectlveness  of  simulated  radar 
Imagery.  Tha  rasults  fallad  to  show  any  algnlflcant  Incraata  In  tha  perceived  mage  quality  or 
tha  parcalvad  training  valua  of  tha  simulated  radar  laagary  produced  frua  tha  aora  datalltd 
transformations. 

Ona  war  labia  Influanclng  tha  fldallty  of  slaulatad  radar  laagary  Is  tha  nuabar  of  polygons  or. 
facas  usad  to  aodal  tha  earth'*  tarraln.  Currant  digital  radar  landaas*  simulator*  (DRLNSs)  raly 
on  digital  data  provided  by  tha  oafansa  Happing  Agency  for  tarraln  elevation  Information.  These 
data  typically  consist  of  terrain  elevations  at  Intervals  of  100  aatars.  A  DRLNS  transformation 
program  fits  polygons  around  thesa  elevation  values  to  depict  tha  terrain.  The  transformed 
terrain  data  are  then  merged  with  surface  feature  information  and  stored  for  use  In  the  real-time 
simulation. 

Ona  variable  In  the  elevation  data  that  Influences  tha  transformation  program  is  terrain 
roughness.  For  perfectly  flat  ground,  roughness  squats  zero,  and  for  maintains,  roughness  can  oa 
greater  than  10.  In  affect,  roughness  measures  tha  rata  of  elevation  change  between  data  posts 
In  a  geographic  region.  Tha  transformation  program  fits  polygons  about  the  elevation  posts  using 
an  Iterative  process  In  which  tha  simulated  tarraln  Is  tested  against  a  criterion  mcisurlng 
transformation  accuracy.  Th<s  accuracy  criterion  Is  Inversely  related  to  roughness.  As 
roughness  Increases,  terrain  accuracy  decreases!  allowable  differences  between  the  elevations  In 
the  source  data  and  the  elevations  of  the  transformed  data  Increase.  Simulated  terrain 
elevations  are,  therefore,  highly  accurate  for  flat  terrain  and  less  accurate  for  rough  or 
mountainous  areas.  This  accuracy  criterion  Influences  tha  number  of  polygons  and,  therefore,  the 
fidelity  or  realism  of  tha  simulated  radar  scene.  Additional  polygons  are  added.  In  an  Iterative 
manner,  until  the  specified  criterion  Is  met.  Because  of  tha  Iterative  nature  of  this 
transformation  process,  the  accuracy  criterion  directly  Influences  the  number  of  polygons  and  the 
length  of  time  required  to  transform  the  digital  tarraln  data.  Since  tha  accuracy  criterion 
Influences  tha  length  of  time  require!  to  complete  the  transformation  process.  It  also  affects 
tha  cost  of  developing  the  simulation.  Therefore,  radar  simulations  of  the  same  area  produced 
using  different  accuracy  criteria  can  differ  significantly  In  both  fidelity  and  cost. 

Because  fidelity  and  cost  are  both  directly  related  to  the  accuracy  criterion.  It  Is 
1^>ortant  to  determine  the  relation  between  the  fidelity  of  tha  simulated  radar  scene  and  Its 
training  effectiveness.  As  Roscoe  (1900)  points  out,  beyond  a  certain  point.  Increases  In 
simulation  fidelity  produce  little  or  no  Increase  In  training  effectiveness.  The  purpose  of  this 
experiment  was  to  determine  tha  relation  between  six  levels  of  transformation  accuracy  and  (a) 
the  perceived  similarity  of  the  six  simulated  radar  Images  and  (b)  the  perceived  training  value 
of  those  Images. 


II,  METHOD 


Apparatus 

A  modified  KC-1J6  DRLMS,  normally  used  for  engineering  development,  was  used  to  transform  and 
display  an  area  40  miles  southeast  of  Knoxville,  Tennessee.  $1x  separate  transformations  of  the 


Figure  4.  Simulated  r4d«r  Iwga  produced  with  wight 
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Figure  5.  S  leu  It  ted  radar  taage  produced  with  weight  ■  4 
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Subject* 

Seven  KC - 135  navigators  froa  tha  Hist  Air  Refueling  Group,  Arizona  Air  National  Guard, 
partlclpatad  In  this  study.  Each  subject  uas  oparatlonally  quallflad  and  currant. 


Procedure 

Subjacts  uara  tastad  Individually  and  askad  to  judge  tha  quality  of  tha  slaulated  radar 
1  Bagary.  At  tha  bagtnnlng  of  tha  axparlaant,  aach  navigator  was  brlafad  on  tha  purposa  of  the 
axperlaant  and  vat  given  a  1:60,000  navigation  chart  of  tha  area.  Each  navigator  was  alto  given 
a  pair  of  dividers  and  a  30  MM  simulated  radar  iBtge  with  an  accuracy  criterion  of  1.0  (Figure 
3).  Tha  navigators  ware  than  askad  to  orient  thenselvas  and  to  Identify  several  proalnent 
terrain  features  to  ensure  they  ware  able  to  correctly  Interpret  the  laaga.  All  subjacts  wara 
able  to  do  this  without  difficulty. 

Following  this  Introduction  procedure,  aach  navigator  evaluatad  the  quality  of  the  30  NM 
simulated  Iwagery  by  perforalng  three  tasks:  preference,  ranking,  and  rating.  Tha  three  tasks 
ware  then  repeated  for  the  16-alle  range.  All  tasks  ware  self-paced  and  tuok  npprorlaately  30 
alnutes  to  conplete.  After  data  collactlon,  subjects  were  asked  If  they  could  offer  any  consents 
or  suggestions  about  tha  oxperlaental  procedures  or  naterlals. 

In  the  preference  task,  each  Isttge  for  a  specific  range  was  paired  with  each  of  the  other 
laages  far  that  range.  Each  pairing  was  presented  twice  In  randoa  order  for  a  total  of  30 
Judgaatitsi  the  position  of  the  laages  was  counterbalanced.  The  subjects  were  Instructed  to 
Indicate  which  laaga  In  each  pair  they  would  rather  use  In  navigation  tasks  and  to  Indicate  a 
preference  even  If  differences  between  the  laages  were  very  slight. 

After  coapletlng  the  preference  task,  subjects  ranked  tha  sis  laages  froa  best  to  poorest 
quality.  To  perfora  this  task,  subjects  were  given  all  six  laages  and  Instructed  to  arrange  then 
on  a  table  until  they  were  satisfied  with  the  order. 

It  is  possible  that  even  laages  Judged  to  be  poorer  In  quality  than  others  way  be  considered 
adequate  for  training.  Likewise,  It  Is  possible  that  alaulated  laages  ranked  as  batter  than  all 
others  way  be  Judged  Inadequate  for  effective  training.  To  test  this,  subjects  were  also  asked 
to  rate  each  laage  on  a  five-point  scale  of  'usefulness  for  Training  Navigators.'  The  scale 
anchors  were  *1.  Excellent!  undlstlngulshable  froa  aircraft  radar,”  and,  *6.  unacceptable!  not 
useful  for  training.' 


111.  RESULTS 

The  preference  data  veto  analyzed  using  a  Thurstons  scaling  approach  that  converted  the  data 
to  an  equal -Interval  scale  of  perceived  quality  (Baird  A  Nona,  1976,  Chapter  7;  Nunnaly,  197B, 
Chapter  2).  The  zero  point  of  this  scale  Is  arbitrary,  and  the  Interval  between  laages  is  based 
on  standard  deviations  coaputed  froa  the  unit  noraal  distribution.  Figure  7  shows  that  for  the 
30  NM  range,  the  three  aost  accurate  transforaatlons,  weight  ■  1,  0.6,  and  0.6,  fora  a  tight 
clustdr.  this  clustering  Indicates  that  these  three  laages  are  not  dlscrlalnably  different  froa 
each  other.  The  results  for  the  16  NM  range  are  different.  They  show  that  the  two  aost  accurate 
transforaatlons,  weights  of  0.6  and  0.6,  are  not  dlscrlalnable  froa  each  other,  and  that  the 
third  aoat  accurate  weight  of  1  It  dlscrlalnably  poorer  than  the  two  wore  accurate 
transforaatlons. 


Range 
30  NM 


Transformation  Weight 

•  4  2  0.8.1  0.6 


0  0.5  1  1.5  2  2.5 


Thurstone  Scale  Values 


Range 
15  NM 


Transformation  Weight 

6  4  2  1  0.8.0.6 


0  0.5  1  1.5  2  2.5 


Thurstone  Scale  Values 


Figure  7.  Thurston*  scalos  of  preference  Oats.  M1|hor  scol*  valuos 
Indicate  greater  preference.  Lower  transformation 
accuracy  weights  Indicate  greater  fidelity. 

The  Man  ranking  data  are  sueMrlzed  In  Table  1.  A  FrledMn  Multi  Saeplo  Tost  (Bradley, 
19.8,  Chsptor  S)  Indicates  no  significant  dlfforoncos  anong  the  Man  ranks  for  th*  thro*  eoro 
accurate  transforMtlons  (alpha  ■  .09)  at  olthor  rang*.  However,  th*  dlfforoncos  hstwoon  tho 
Man  ranks  for  those  transformations  and  th*  next  transforectlon  level.  2.  was  significant  (£< 
.09)  for  both  rang**.  Those  results  Indlcot*  that  th*  thro*  eost  accurate  transforMtlons  woro 
not  perceptibly  dlfforent  fron  each  other  and  that  the  next  level  was  consistently  ranked  as 
lower  In  quality. 


Table  1.  Moan  Rankings  of  iMg*  Quality 


Rang*.  HR 

9 

4 

2 

1 

0.9 

0.9 

30 

9.9 

9.4 

3.9 

3.0 

1.9 

1.9 

19 

9.9 

9.4 

3.7 

2.9 

2.0 

1.4 

The  Man  rated  training  valu*  of  each  1waga  Is  shown  in  Table  2.  Those  data  wore  analyzed 
using  a  one-way  analysis  of  variance  (Minor,  1971,  Chapter  4).  Those  analyses  show  that  the 
ratings  assigned  to  th*  thro*  MSt  accurst*  transforMtlons  war*  not  significantly  different  fron 
oach  other  for  either  range  (alpha  ■  .09)  and  that  th*  Man  rating  for  these  iMges '  was 
significantly  higher  than  th*  next  eost  accurate  transforMtlon  level. 


Table  2.  Mean  Ratings  of  Usefulness  for  Training  Navigator* 


Weighting  Factor 

Range,  NN 

6 

4  2  1  0.6 

0.6 

30 

4.7 

4.6  2.9  2.4  1.6 

1.6 

16 

4.9 

4.9  3.7  2.6  1.9 

1.7 

IV.  DISCUSSION 

Tht  prosont  projoct  tddrottod  two  major  questions.  The  first  question  dealt  with  the  level 
of  fidelity  or  realise  that  Is  necessary  to  produce  an  adequate  slaulatlon.  In  this  study. 
Increasing  fidelity  was  achieved  by  Increasing  terrain  vertical  accuracy;  1.*.,  changing  the 
accuracy  criterion.  Increasing  transformation  accuracy  significantly  Increases  the  processing 
tlee  necessary  for  the  ORINS  to  transfore  digital  data  base  tnforeatlon  Into  a  simulated  radar 
leage.  Coepared  with  the  standard  transforeatlon,  weight  ■  1,  the  most  accurate  transformation 
with  weight  •  0.6  required  2SS  more  processing  ties  while  the  least  accurate,  weight  ■  6, 
required  39*  less  time.  The  data  obtained  from  the  preference  and  ranking  tasks  indicate  that 
transformations  with  weights  of  1,  O.S,  and  0.6  are  perceptually  similar  tu  one  another. 
Therefore,  It  can  be  concluded  that  Increasing  the  fidelity  of  the  simulation  by  using  a 
weighting  factor  less  than  1.0  products  no  significant  gain  In  tht  perceived  quality  of  tha 
simulated  Image. 

The  second  question  addressed  by  the  present  study  concerns  training  effectiveness.  The 
preference  and  ranking  tasks  assess  the  dlscrlmlnablllty  of  the  stimuli  but  not  their  usefulness 
for  training.  The  rating  data,  however,  show  that  the  three  more  accurate  transformations  are 
judged  to  hav6  equal  training  value  and  to  be  significantly  more  useful  than  the  less  accurate 
images.  According  to  Roscoe  (1960),  Increasing  realism  beyond  some  point  can  Increase  cost 
without  Increasing  training  effectiveness.  These  results  support  Roscoe's  assertion.  For  both 
ranges,  radar  simulations  produced  with  normal  accuracy,  weight  ■  1,  (a)  are  Indistinguishable 
from  more  accurate  simulations  with  weights  of  0.6  and  0.6,  (b)  have  the  same  training  value  as 
the  more  accurate  transformations,  and  (c)  can  be  produced  at  significantly  lower  cost.  Although 
the  training  effectiveness  of  each  level  of  simulation  fidelity  cannot  be  tested,  a  reasonable 
assumption  Is  that  If  operational  navigators  cannot  reliably  distinguish  between  the  various 
lavels  of  transformation  accuracy,  then  these  simulations  should  have  approximately  the  same 
training  value. 

The  subjects  were  asked  to  comment  on  the  experimental  materials  and  procedures.  The 
altitude  of  the  stimulus  Images,  10,000  feet,  was  consistently  mentioned  as  a  problem.  These 
subjects  were  navigators  on  KC- 135  aircraft,  which  normally  fly  at  altitudes  above  30,000  fset. 
Other  aircraft,  such  as  8-b2s  or  FB-llls,  typically  fly  at  less  than  1,000  feet  above  the  highest 
local  terrain.  Th»  stimulus  Images  produced  at  10,000  feet  are  therefore  not  representative  of 
nearly  all  Air  Force  navigation  environments.  Since  the  effect  of  transformation  accuracy  on 
Image  quality  Is  more  pronounced  at  low  altitude  than  at  high  altitude,  a  level  of  transformation 
accuracy  which  produces  acceptable  Images  for  low  altitudes  will  also  produce  acceptable  Images 
for  higher  altitudes.  Before  a  final  decision  is  made,  the  present  experiment  should  be  repeated 
using  lower  altitude  raaar  Imagery  and  navigators  familiar  with  low  altitude  missions. 
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